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THE EVALUATION OR DEVALUATION OF CURRICULUMS REQUIRES 
THE VALIDATION OR INVALIDATION OF THEORY AND IS NOT 
DETERMINED BY PHILOSOPHICAL DISPUTE. AS LONG AS A CURRICULUM 
IS CONSIDERED TO BE A CLOSED INFORMATIONAL SYSTEM, INTERNAL 
CRITERIA MAY HELP ITS SEQUENCE AND ESTABLISH ITS CONSISTENCY 
BUT CANNOT VALIDATE IT. TO EVALUATE A CLOSED INFORMATIONAL 
SYSTEM, THE RELEVANT INFORMATION WHICH PREVAILS OUTSIDE THE 
SYSTEM MUST BE DETERMINED AND TESTED AGAINST THAT WHICH 
PREVAILS INSIDE, AND ITEMS WITHIN MAY THEN BE DISPLACED OR 
RETAINED. THE OUTSIDE INFORMATION REPRESENTS AN EXTERNAL 
CRITERION OF THE INSIDE INFORMATION. OTHER CRITERION PROBLEMS 
SUGGEST MODIFYING THE CONCEPT THAT A CONTROL GROUP MUST BE A 
NAIVE PLACEBO GROUP, AND REEXAMINING STATISTICAL DECISIONS 
SUCH AS WHAT CONSTITUTES ADEQUATE EVIDENCE OF THE IMPACT OF 
THE CURRICULUM. WITH CURRICULUMS IN VARIOUS STATES OF 
TRANSITION, INCLUDING CHANGING METHODOLOGY, ADOPTING NEW 
OBJECTIVES, AND INCORPORATING NEW INFORMATION, THE PROBLEM 
BECOMES ONE OF EVALUATING AN OPEN AND DYNAMIC SYSTEM. HERE, 

THE FIRST PROBLEM IS TO DETERMINE THE BOUNDARIES, NOT BY 
DEFINING CHRONOLOGICAL AGE, BUT BY CREATING SOME OPERATIONAL 
DEFINITION OF VOCATIONAL ASPIRATIONS. LONGITUDINAL STUDIES 
ENTAILING BOTH THE PREDICTION AND ASSESSMENT OF CHANGE, 
INDEPENDENT OF EACH OTHER, ARE NECESSARY FOR SUCH CURRICULUM 
EVALUATION. THIS WILL PRODUCE CRITERIA FOR STABILITY AND 
CHANGE. RESPONSES TO THE PAPER BY RALPH E. MASON AND ROBERT 
M. WASSON ARE INCLUDED. THIS PAPER WAS DELIVERED AT THE 
NATIONAL SEMINAR FOR RESEARCH IN VOCATIONAL EDUCATION 
(URBANA, ILLINOIS, NAY 16-20, 1966). (EM) 
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Criterion Problems in Curriculum Research 
Donald A* Leton 

Education Research and Development Center 
UhiVersity of Hai/aii 

One day, about 160 years ago, the French mathematician Laplace (1812) 
computed the probability that the sun would rise on the following day* 

L. 

Based on his assumption that the earth was created about 3,000 years 
previously, and that the sun had risen for 1,826,230 consecutive days, he 
was able to arrive at a finite probability that the sun would rise again 
the following day! Manyyears from now an educational historian very likely 
will review the educational events of our times. He will uncover a veritable 
mountain of writings from books, journals, convention programs, seminars, 
colloguia and symposia. In viewing the surface of this mass he might be 
led to the conclusion that education in the 20th century was a very complex, 
thorough and scientific discipline. ^7hen he proceeds however to analyze 
the concepts contained in all these documents he x^ill find a great deal 
of repetition. In fact, if he were to classify the ideas that had been 
repeatedly professed and expounded up to May 16, 1966, and weigh their 
redundancy in the literature, then his prediction would obviously be that 
the sun will rise and set today on the same ideas that prevailed yesterday. 
The probability that an educator will express a new idea today which might 
move this mass, or even give it a different orientation, is very remote. 

And tomorrow this paper also will be only a small conglomerate in the same 
mass. If in reading this paper, or in attending this seminar you are 
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expecting to learn new concepts, or new ideas which might potentially 
revolutionize vocational education then you may be disappointed. These 
preliminary comments are not just an apology; they are also intended to 
describe the nature of our discipline. T^ether we consider the schools 
in our nation, in European nations, or even in the underdeveloped nations 
in the process of installing educational systems., we find within them an 
amazing perseverance of yesterday's practice. 

It's not the intention of this paper to devaluate the stability of 

curricula, nor is there much that can be said in its praise. The purpose 

of the illustration is to create an appreciation for a difficult problem 

in curriculum evaluation, that of innovation. Perhaps a better appreciation 

of this phenomenon is expressed in a limerick composed by a friend, Nolan 

C. Kearney, who was a very serious ci^rriculum scholar. 

There once was a teacher named Flo 
Thought progress exceedingly slow! 

She was stuck in a rut 
Clean up to her butt 
In the morass of status and quo. 

And so the problem is defined. The curriculum is identified as an 
obdurate structure, bound by the habit patterns of many persons. There 
are educators who teach the same concepts and profess the same ideas as 
were taught generations ago. There are administrators who attend the same 
service clubs and carry out the same kind of situational management as their 
predecessors. There are the local P.T.A. 's or other parent groups that 
generally declare their support of schools and their confidence in school 
policies. Occasionally some parent groups become apprehensive that the 
traditional values of the 3 R's and other academic standards might be lost 
because of modem innovations. They may even become reactionary and request 
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that the previous curriculum be restored! IThether they petition for a 
change 9 or for malnter^ance of the status quo, they seem to have little 
Impact on the day-by-day affairs of the school* 

There ate the school architects vho draw the same four walls and 
corridors^ There are the teachers whose courage for change Is somewhat 
exceeded by the security of time-tested practice and well-charted courses* 

And there are also the curriculum supervisors* Their mode of functioning 
has been to mimeograph and distribute the curriculum "guides" that are soon 
shelved in a department office or a curriculum library* 

Perhaps this description Is not entirely accurate* It may sound unfair 
to modem education^! To the extent that It Is accurate, hmiever. It defines 
the curriculum as a very stable phenomenon within a closed system* 

Have educational researchers ever considered how to evaluate a stable 
phenomenon In a closed system? Have we ever faced this task? Later in 
this paper the curriculum will also be described as a very dynamic process 
in an open system; and the contingent evaluation problems will be reviewed* 
But, for the moment, let us consider the traditional In vocational education* 
The social utility and economic value of vocational education represent two 
Important philosophical bases* Since these are philosophical they can 
be challenged by the Socratlc philosophy of the personality development and 
cognitive shaping of the student* In contrast to this, the use of Instruc- 
tional models and of Imitation learning In apprentice training are charac- 
teristic of traditional vocational education and they can be readily 
defended on philosophical as well as psychological grounds* Philosophical 
debates, hcn^ever, will never establish the validity of curricula* The 
philosophy of liberal and general education may challenge vocational education; 
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and the cultural and technological advances nay stress It, but the choice 
o£ philosophical values In education Is always open to debate; and generally 
the strongest voice wins. 

The evaluation or devaluation of curricula requires the validation or 
Invalidation of theory » and Is not determined by philosophical dispute. 

There Is as much need for theoretical grounding of vocational education as 
there Is for other school curricula. One of the popular theories about 
curricula Is that they represent Informational systems. Unfortunately the 
limitations of this theory are not generally recognized as yet. Presume 
that a sequence of courses, e.g., the courses to become an electrician or an 
electronics technician, are established on the basis of Informational theory. 
The teachers convey Information, the textbooks contain Information, the 
blackboards convey Information, and the student acquires Information. The 
acquisition of that Information Identifies him as an electrician. HC may 
even take state board examinations which test for the amount of Information 
from that curriculum. As long as that curriculum Is a closed system, then 
the Information within It is both necessary and sufficient. There Is no 
way of discarding any which may be obsolete, superfluous, or Inaccurate. 
Internal criteria may help to sequence the information within the curriculum 
and may also establish its consistency, but It can not validate It. 

In order for educational research to qualify as research It requires 
that a theory may be either refuted or confirmed. To evaluate a closed 
Informational system the relevant Information which prevails outside the 
system must be determined and tested against that within It. Items of 
Information within the boundaries of the curriculum may then be displaced, 
or they may survive. The value of the information outside the system 
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represents an external criterion of that T7ithin it* 

The folloi?ing syllogism illustrates the problem of evaluating a closed 
system: 

1. The best vocational educators etoployed in Wisconsin are graduates 
of Stout Institute* 



2* Most of the vocational educators employed in Wisconsin are graduates 
of Stout Institute* 



Therefore Wisconsin employs most of the best vocational educators* 

Some of you may have already decided that the conclusion is false* 

What we need to recognize, ho^jever, is that on the basis of the information 
given the truth or falsity of the conclusion can not be proven* 

Perhaps you may ask for the relevance of this for curriculum evaluation* 
Several years ago I had the privilege of working with Guy Buswell on an 
evaluation of ttio methods of teaching reading (1964)* In spite of the fact 
that the methods V7ere theoretically quite distinct the reading achievement 
of the pupils taught under the tc^o methods was not significantly different* 
As the applications of the two methods t;ere observed in classrooms, it 
became apparent that they did not result in uniquely different programs* 

The theoretical formulation of the methods was studied and interpreted by 
teachers on the basis of their concepts, and their Judgments about the teach** 
ing of reading* These were then effected in classroom practice and finally 
culminated in pupil experiences* The variation in classroom practices 
among the teachers in a given method was far greater than the variation 
between the two methods* I^at originated as a curriculum study eventuated 
with the assessment of pupil achievement occurring under similar classroom 
experiences* 
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The study of the comple^s relationships between student learners add 
their curricula Is in Its Infancy , both theoretically and enq^lrically. In 
Its theoretical beginnings only a few concepts relating the two have been 
Identified* E:camples of these would be Piaget's theories of conceptual 
and quantitative development (Flaveli, 1963), and Gagnd’s (1962) theory 
of hierarchical behavior performances* The validation of concepts from 
theory requires that the researcher operates on both the theory and empirical 
level. At the empirical level the researcher faces a decision as to what 
constitutes adequate evidence of the influence of the curriculum on the 
individual. 

Perhaps at this point it might be well to consider the types of criterion 
problems which are met in educational research. In discussing the informa** 
tional content of the electrician's curriculum the need for an external 
criterion for validating the content was illustrated. A second criterion 
problem pertains to criterion groups. In the usual design of experiments 
individuals are randomly assigned to either a treatment or a control group. 

A number of researchers (Cronbach 1964, McKeachie 1963) have argued that 
control groups have limited value. These arguments are based on the assuiiq»<* 
tion that students and teachers are usually aware of their experimental or 
control group membership and then may extend personal energies beyond normal 
performance to affect the outcomes. On the other hand most researchers 
recognize that discarding the control-group design and reverting to a single 
experimental group, pre- and posttest design, present a greater hazard of 
erroneous conclusions. It is. probably better to modify the concept that 
a control group must be a naive placebo group. 

Criterion groups can also be subgroups of students which differ from 
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each other on Imovjn characteristics. As such they are not representative 
of the entire population but rather of only a specified segment in which 
particular effects might be hypothesized, e.g., girls vs. boys^ bright vs. 
average, etc. !The lcnov<7ledge gained from assessing the effects of curricula 
on criterion groups may be of much greateii: value than assessing its effects 
on a heterogeneous group representative of the population. The subsequent 
Job performance of various types of students can also be a criterion of 
the interaction between curricula and students. 

The next type of criterion problem which needs consideration in 
curriculum research is the statistical criterion; The question as to what 
constitutes adequate evidence of the impact of the curriculum is one example 
of a statistical decision. 

In one curriculum experiment the students in an experimental curriculum 
achieved as well on a posttest as did the students in the traditional 
curriculum. If the experiment had ended there, then any decision about 
the superiority of either curriculum would have to be suspended, or be 
based on prior evidence. A factor analysis of the criterion test however, 
revealed that the achievement should not have been regarded as a single 
dimension; it contained four dimensions. The traditional students were 
superior on one or ti^o of the dimensions, for example, subject information; 
and the experimental students were superior in another dimension, e.g., 
concept mastery. The statistical test of composite achievement led to the 
conclusion of equivalent levels. The analysis of the criterion instrument, 
hoTjever, led to the conclusion that the curricula resulted in different 
types of achievement. At this point we should add that neither of these 
statistical conclusions valuate or devaluate the curricula. Since the 
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experiment was Initially designed to test for the superiority of composite 
achievement the factor analytic information about the types of achleven^nt 
was secondary* Some of the curriculum sellers and the curriculum consm^rs 
might choose certain types of outcomes as being more valuable* This, 
hoiirever, should be recognized as subjective preference rather than curriculum 
science* Scientific procedures require that concept mastery and subject 
information be further tested against later criteria, for example, job 

•i 

performance* 

Curriculum research has typically studied the effects of educational 
programs using significance tests on the mean differences appearing In 
posttests* Practical limitations have generally precluded the use of 
extensive variance and covariance designs* The possibility of sequential 
analysis, using small samples of students within a local curriculum, for 
decisions about pilnor effects should be considered. Although a number of 
educational researchers question the usefulness of true experimental l^gigns 
for curridulum problems, nevertheless, the definitive answers in validating 
curricula procedures must be reached through such designs. 

It appears however that, at the present stage of developmc^nt in educa* 
tional research, adequate definitions of curriculum variables, curriculum 
objectives, and student characteristics are not available. Hence they 
cannot provide a guide for disciplined and relevant research* For this 
reason more extensive use of multivariate analyses, such as factor analysis, 
canonical correlation, and discriminant analysis, is recommended to help 
identify pertinent variables and theoretical constructs* For example, a 
curriculum developer may aspire to objectives such as problem solving, 
scholarly interests, and motivation* These represent a wide array of 
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student behavior* Belore such outcomes can be ascertained In curricula they 
need to be defined In terms of their simple components and correlates* 

The multivariate procedures referred to can serve in the analyses of complex 
variation within teachers, curricula and students and of the complex 
relationships among them* 

A conspicuous aspect of recent educational theory has become identified 
theory * This term does not indicate an integrated system 
of scientific axioms and theorems, not even to the degree of other theories 
in education* I4any educators would regard Instructional theory as inseparable 
from learning theory and place them both within curriculum theory* Others 
may be more inclined to annex instructional theory to curriculum theory 
and thereby justify the usual lin!;age of curriculum X'jith instructional 
functions* Instructional theory consists of a type of Inquiry, and although 
it might npx7 be difficult to establish its boundaries among other educa* 
tlonal theories, it seems to be identified with detailed questions rather 
than xfith definitive answers* 

The scientific discipline of education has, for the greatest period 
of its history* managed without Instructional theory and did not feel its 
lack* The present generation of Instruction researchers are largely engaged 
in eliciting brief verbal or performance responses, e*g*, key pressing, 
in student subjects whose sensory attention is being controlled by particular 
kinds of visual and auditory stimuli* Passing attention is sometimes given 
to motivation variables in this research, but generally the artificial 
motlvatii;n xjhich is generated by verbal directions and mechanical devices, 
especially x^hen these represent a diversion from the usual classroom instruc- 
tion, is accepted as part of the inatructional stimuli* The approach h a s 



been to modify the broad concepts of curriculum and to substitute a more 
restricted concept of instruction* It is apparent, hoi^ever, that instruct 
tional research is the application of stimulus<*response psychology to the 
field of education* Important as this approach may have been in the history 
of psychology, the same limitations T7hlch it held for the study of general 
behavior are noi -7 apparent in the study of school learning* It appears that 
instructional theory and its related technology T7ill fall short of the 
educators' expectations because it does not deal with the response domain 
of broad educational objectives* 

Previously in this paper the curriculum vas described as a stable 
system, and relatively closed to external influence* Validating the 
informational content of such a curriculum, was illustrated by establishing 
an external criterion of information* Such a curriculum might also be 
evaluated through descriptive and manipulative procedures* Variables within 
the curriculum such as the number of hours in shop application, the amount 
of homework, or the number of students per class can be manipulated until 
they produce a demonstrable effect*. In such studies the internal variable 
changes must be maintained until a reliable description of the generated 
effects can be made* Hany curriculum developers, hot^7ever, resist the 
internal manipuhtlon of variables with the view that all the relevant 
elements should adhere in a unified organizational structure and that the 
effects on students are due to the total Influence of the curriculum rather 
than its separate elements* 

Students may also be shifted from one stable curriculum to another* 
Shifting students from one curriculum to another, however, produces other 
research hazards* For example, the student-effects from the first curriculum 



ll 

«> 

nay not be reversible by the second^ Another possibility is that the 
end result of the second curriculum is due to the cumulative or combined 
effects of both placements » i#ey, the curricula are interactive rather 
than independent# 

There is fairly consistent evidence that a student's previous educa- 
tional history is a more important determinant of his future performance 
than are his present aptitudes# This seems to indicate that the student 
also functions as a stable system# Unfortunately many neu curricula are 
being developed with the philosophical principle of educational reductionlsm* 
This principle holds that a student's characteristics are entirely determined 
by the experiences he has had# or will have, in his curricula. This can 
be constrasted with one of the principles of progressive education# For 
example, Rugg's intetest^centered classroom is based on the principle that 
the curriculum is or should be determined by the student's characteristics; 
in this case his interests# It is not vsy intention to debate the contrasting 
philosophical principles but rather to refer to the orientations which 
they bring to curriculum theory and to the subsequent research# Curriculum 
evaluation must consider the stability and modifiability of the student to 
accomplish its purpose# Many of the dynamic qualities of adolescents and 
youth, for example, are not adequately luiown for controlled research in 
vocational education# Vocational aspiration and vocational Interests may 
be regarded as either an Input or an outcome of vocational curricula# The 
Interaction bett/een personality traits and vocational interests, and 
determinants of vocational choice are only in the theory stage (Super et al 
1963)# Intellectual maturity in terms of vocational aptitudes may be 
presumed for incoming vocational students, or may be challenged and tested 
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by the currlculuii£ itself. 

1 

Since World War XI the education profession has roused itself to 
face the picture of its oim stagnation and obsolescence. Perhaps it would 
be more accurate to state that the press of national and world affairs has 
forced a confrontation of educators with outmoded curricula. One of the 
outcomes of this in vocational education has been the development of 
retraining programs. 

Unfortunately when academicians ip the subject matter fields became 
aware of the problem they hurriedly developed a wide variety of programs 
and materials; and the federal government has spent millions of dollars 
in implementing these in the public schools without any orderly and systematic 
procedures for evaluation. The need for evaluation has only been a recent 
hindsight. In fact professional educators have not as yet identified the 
manner in which traditional curricula were outmoded; nor the scope of 
their inadequacy. A curriculum can be identified as outmoded when its 
theories, methods, and content are less effective in producing acceptable 
outcomes than are other available curricula. 

In some school systems there have been wholesale changes without 
regard for teacher preparation or for possible student effects. The 
modem curricula were characterised by reorganization of the content and 
by modification of its presentation. Curriculum administrators recognized 
that a good way to avoid public criticism was to change; and changes were 
made solely for the sake of change.. Teachers who were teachers of the 
"modem math" or the modem science assumed an emlted status among their 
peers, without any real consideration of their effectiveness. High schools 
presented the modem curricula without foreseeing the disadvantages its 



students would eventually face in college entrance examinations and In 
traditional college courses* 

In a relatively short period of time the closed system of education 
was open to influences from political sys terns » e.g*, the federal government; 
from social systems and their representative organizational groups; and 
from the economic system^ with revolutionary changes In industry and 
business producing drastic changes in manpower and training needs* 

Internal changes also produced dynamic and unstable curricula* Changes 
In teaching methodology^ instructional materials and procedures, grading 
practices, and curriculum "standards*' were concurrent with other changes* 
Perhaps it would be more accurate then to define education as a number 
of systems in transition states* The student is in transition from child- 
hood to adulthood* His cognitive abilities are in various stages of maturity 
and immaturity* His social attitudes and personal characteristics show 
day-to-day fluctuations as well as progressive developmental states* 

The curricula are in various states of transition from yesterday's 
practices to tomorrow's eispectancies* Changes are seen in teaching 
methdology, in the incorporation of new information and in the adoption 
of nei'7 objectives* Uhether we accept such states as desirable or undesirable 
is irrelevant; the major question is hcx7 they are to be evaluated* Inasmuch 
as they are transitional they may also be expected to shoir some cyclic 
bacla-7ash, and irregular variability* 

The first problem in studying a dynamic state Is to determine Its 
boundaries* To illustrate this for vocational education we might regard 
adolescents and youth as being in a transitional state In terms of their 
vocational aspirations* The boundaries of this state are not necessarily 
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defined by chronological asds» but rather by some operational definition 
of vocational aspirations. Aspects of vocational guidance and vocational 
curricula nay then be set to correspond to these boundaries « 

lie night also vie^^ vocational curricula as being in a state of 
transition. The boundary of the need for blacksnithS in the horse and 
buggy days was defined by the nass production of autonoblles. The subsequent 
need for auto nechanics has ttow been supplenented by the need for jet 
nechanics. Technicians who are in such great denand for defense and defense- 
related industries today nay soon require retraining for peacetine applica- 
tions of their skills. The present curricula for industrial technicians 
nay be nade obsolete by a future denand for space technicians. Perhaps 
curriculun developers should endeavor to foresee such changes and develop 
evaluation plans prior to wholesale revisions. 

VJe night also ask at t^hat point does a curriculum change from transi- 
tional to stable? The quantification of curricular change is definable 
by rates of changes among teachers and by variation in successive groups 
of students. For some teachers there may be little or no course revision 
after the second time he offers it. Such a point in transition nay be 
identified by successive measures of student outcomes. The dynamic and 
open features of curricula may only be intermittent and shot; a system lag 
to historical events and to social pressures for change. Obviously, longi- 
tudinal studies which entail both the prediction and the assessment of 
change, independent of each other, are necessary for such curriculum 
evaluations. 

In summary this paper has been written to try to convey some of the 
problems in curriculum research. The need for theoretical constructs 



15 



relating curriculum variables to student outcomes was pointed out. Exan^les 
of external and internal criteria, the use of criterion groups, statistical 
criteria, and criteria for stability and change were given. Curricula were 
descvTibed as stable closed systems and as open transition states. Procedures 
for the evaluation of these for the future engineering of education were 
suggested. 
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The need for curricular innovation and evaluation in vocational 
education as pointed up In Or. Leton's paper deserves our attention and 
concerted efforts. Supervisors and educators are working overtime to 
implement new programs making use of the project method (called in-school 
in some states and Intensive laboratory in others) and/or the cooperative 
method of instruction as capstone educational offerings to programs of 
instruction in bur various vocational areas. . The development of nev/ 
programs under the impetus of the Vocational Education ,\ct of 19o3 is 
Just now gathering momentum. These prospects of development and grc?wth 
call for reflection and evaluation. 

Or. Leton observes that a curriculum can be identified as outmoded 
when its theories, methods, and content are less effective in producing 
acceptable outcomes than are other available curricula. Our research 
must investigate these other available curricula and at the same time 
evaluate our existing curricula. 

What better criteria might be used for this evaluation and research 
than the sixteen fundamental principles of guality vocational education 
as set forth by Prosser and Quigley. * These same underlying principles 
still pertain. They are not outmoded by the passage of time. 

For example, vocational education will continue to be efficient in 
proportion as the environment in whicli the learner is trained is a replica 
of the environment in which he subsequently must work. Vocational education 
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will be effective in proportion as it trains the individual directly and 
specifically in the thinking habits and the manipulative habits required 
in the occupation itself. And, it must be realized that every occupation 
has a body of content which is peculiar to that occupation and which 
practically has no functioning value in any other occupation. But, there 
are commonalities among related occupations within job clusters. And, 
there are certain commonalities among related occupations for which the 
various disciplines in vocational education may well discover ways of 
sharing their particular know-how in these areas with other vocational 
discipl ines. 

Business and distributive education are striving to determine the 
elements of skills, knowledges, and understandings common to most of the 
business occupations in order to assure that curricular offerings include 
these basic elements. The extent to which these common elements can be 
found and taught will determine the degree to which training can be 
supplied for clusters of occupations to assist in preparing students for 
mobility in related occupations. The underlying elements of management, 
marketing, recording, and information processing skills involved in 
business occupations must be taught to students having career objectives 
in business occupations. Many students with career objectives in business 
occupations are located in business or distributive curricula. Other 
students with career objectives iii business occupations are located in 
home economics, industrial education, or agricultural curricula, where 
the technologies of these areas are so important to the understanding of 
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product information and the successful dealing with customers. Hav can 
vocational educators best serve these students by sharing program offerings, 
teaching specialties, and technological know-how? 

When the principles of vocational education referred to above are 
translated into a cooperative method approach to instruction a certain 
pattern emerges. At the moment, certain elements are considered essential 
for a successful cooperative program. Essentially, the cooperative plan 
of vocational instruction uses the work situation as a "school laboratory" 
in which occupational competencies are developed through supervised 
occupational experiences while related instruction is given in school. 

The cooperative plan actually represents a tripartite instructional 
situation. Thinking of the various psychomotor, cognitive, and attiudinal 
learning which go to make up the package of competencies required in any 
given occupation: 

1. Some are learned solely in school in the related class and 
applied on the Job. 

2. Some are learned solely on the Job. 

3. Some are learned partially in school and partially on the job. 

in addition, the career objectives of the student make it mandatory 

that certain things, such as how to train and how to supervise, be learned 
in school even though they are not required in the student's training 
station Job. Further, the cooperative plan also gives emphasis to discussion 
in school of problem situations encountered on the job, such as human 
relations involving how to benefit from supervision and constructive critism. 
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In order to- carry out tbs vocational Instruction function just 
described, the cooperative plan involves the following elements, as far 
as we know now: 

1. Related instruction in school. 

a. Basic related instructions 

b. Specific related instruction. 

2. Selected training station. 

3. Student** learner with a career objective. 

4. Preparatory curriculum. 

5. Step-by-step training plan. 

6. Adequate on-the-job supervision. 

7. A qualified teacher-coordinator. 

8. Adequate coordination time. 

9. Suitable classroom facilities and instructional materials. 

10. Well defined school policies regarding the program. 

11. Well organized program records. 

12. Use of an advisory committee. 

Our best cooperative programs have been evaluated up to now on 
the basis of the extent to which these elements have been perfected and 
the extent to which graduates have been placed in and have been successful 
In the occupations for which they have been trained. Are there other 

important elements to be considered? Are certain of these elements super- 
fluous? 
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The in*school laboratory method of t'?achsng for office occupations* 
home economics, industrial education, and agriculture has been used for 
some time. However, too often these curricular offerings are considered 
isolated courses. Hew approaches emphasize a program approach with the 
intensive laborocory, as well as the cooperative arrangement, becomes the 
capstone to a series of prescribed courses and experiences. In business 
education, advanced level regular business and distributive subjects may 
provide for intensive experience through the medium of a laboratory not 
only in which exercise and problems involving the application of knowledges 
and skills will be an integral part of the learning experience, but also, 
in which the instruction will center upon the performance of production 
tasks on actual work projects which are brought Into the laboratory. 

Programs terminating with instruction in the intensive laboratory 
include a planned sequence of courses and make specific provisions for 
the following mini mums: 

1, Satisfactory dc\/elopment of basic business and economic 
competencies so necessary for entering and being successful 
in business occupationa. 

2. An understand: ng of business principles which includes the 
theory, principles, and practices of administration and 
operation. 

3* Instruction in background business and economic information 
essential to employment. 
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4. Development of skills sufficient to indicate an aptitude for 
and an interest in pursuing the iniensive training of the 
laboratory . 

5. Practical applications of skills, knowledges, and techniques 
requisite to success and advancement in the enrol lee's career 
objective. Relevant integrating experiences are provided 
through the intensive laboratory. 

Are these the most important minlmums? Are there others? .to the 
intensive laboratory approach to occupational preparation just as effective 
or more effective than the cooperative method? Is the cooperative method 

superior? Or, should both methods be utilized by school systems that can 
support them? 

In this short reliction to Dr, Leyton's presentation, questions 
have been posed that need answering as a search is made for improvement 
In our approaches to vocational education. Let us take the best we 
have learned from applying the underlying principles of vocational education 
revealed from our experiences of the past and revise and supplement our 

curricula of the future on the basis of valid experimentation, research, 
and evaluation. 

^ Grosser, Cnsries A. and Thcma?; H. Q,uiq1ey. ‘Voc at iona* 

Effiication in a Democracy . Chicego; American TechnicaTlocfetY; 1349. 







National Seminar on Curriculum Evaluation in Vocational Education 
University of Illinois, Urbana, Illinois, May 1966 

Criterion Problems in Curriculum Research: A Reaction 

Robert M* V^asson 
College of Education 
University of Iowa 

Professor Leton*s paper has placed me in a difficult dilemma* I agree 
heartily, vehemently, and with more than a little passion with his deliniation 
of the problem confronting work in curriculum research* I find, however, his 
suggested solution to be characterized by many of the same conditions and 
problems that he points out as existing in the difficult problem of obtaining 
curricular change* 

Let me illustrate this contention* Professor Leton rightly points to the 
difficulty of evaluating a closed system and to the necessity of obtaining 
data from outside that system to evaluate the behaviors within the system 
he describes* In doing this our friend establishes a new closed system* On 
page four he has indicated that and I quote **in order for educational research 
to qualify as research, it requires that a theory may be either refuted or 
confirmed*” Immediately he has created a closed system by which educational 
research or ”true research” is defined. Ue are left vjith the alternative of 
accepting that system or we may choose to define research quite differently* 

Me may argue that the development, the description, or the deduction of new 
information or new bits of data is that which characterizes research and that 
indeed, research so narrowly defined as the confirmation or refutation of a 
theoretical position severely limits, if not almost destroys, the possibility 
of substantial advances in the area of research in curriculum whether it be in 
vocational education or in the preparation of educational researchers* I 
would wonder if the Harvard psychologist, B* F* Skinner, could be considered 
aresearcher* I certainly would consider him so and yet Skinner has strongly 
maintained for a number of years that he is atheoretical in approach. In fact. 
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Skinner has maintuined in his book Cumulative Record that the ref^archer 
is the one who is willing tp drop or abandon a theoretical position 
to pursue an interesting piece of behavior in whatever direction it 
night lead him* It has become indeed fashionable for educational 
psychologists, to imitate the experimental psychologist (as the 
experimental psychologist imitate the behavior of physicists of a decade 
ago) in demanding theoretical propositions which would lead to testable 
hypotheses which would in turn lead to conclusions about behavior. 

Marvin Dunnette in a very x'ecent paper in the American Psychologist 
has labeled such theory building as fashion and folderol in Psychology 
and, in Ounnette's words, 

"Fashions— those manners or modes of action taking on 
the character of habits and enforced by social or scientific 
norms defining what constitutes the "thing to do"— included 
theorizing and tiieory building, criterion fixation, model 
building, null hypothesis testing, sensitivity training, 
being productive at vjork, developing authentic relation- 
ships, devising "cute" experiments, simulation, using 
"elegant" statistics, and so on. 

Folderol— those practices characterized by excessive 
ornamentation, nonsensical and unnecessary actions, 
trifles and essentially useless and wasteful fiddle- 
faddle— included tendencies to be fixated on theories, 
methods, and points of view, conducting "little" studies 
with great precision, attaching dramatic but unnecessary 
trappings to experiments, asking unimportant or irrele- 
vant questions, grant smanship, coining new names for 
old concepts, fixation on methods and apparatus, seeking 
to "prove" rather than "test" theories, and myriad 
other methodological ceremonies conducted in the name 
of rigorous research." 

An example of this particular form of behavior certainly includes 
the fixation on highly con^lex statistical analyses including variance, 
discriminant, canonical, and factor and the application of such 
operations to data frequently not worth the scouting which is employed. 
Inevitably, this does lead to the plethora of publications and papers 
referred to quite caustically and with cause by Professor Leton. Such 
verbal fecundity defies the capacity of any individual or- even 
research institute to comprehend them. 
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It is worth noting at this point that many of the publications 
which appear in such journals as the Journal of Experimental Psychology ^ 
the Journal of Applied Psychology, the Journal of Personality and 
Social Psychology, and the Journal of Abnormal and Social Psychology 
are, for the most part rigorously designed, if dealing with amazingly 
inconsecpiential pieces of behavior. The conclusions the naive reader 
would draw from reading articles from these journals is that psychology 
has made numerous and important advances. This reader could not help 
but be impressed as he scans complex analysis of variance designs 
^ liberally dotted with one, tVifo, or with a blessed smile three asterisks, 
indicating indeed significance has been obtained. The sophisticated 
reader is perhaps somev;hat disturbed when he examines the magnitude 
of the relationship between the independent and dependent variables 
so investigated. In a recent investigation by Milton HackeZ (1966) 
studios in these journals were randomly selected and the F ratios and 
T tests were converted to correlation ratios etas in attempt to esti* 
mate the degree of association between the independent and dependent 
variables. Of the total 112 correlations ratios formed over one-third 
failed to reach .30, with the median value of all 112 being only •42. 

It is disturbing to the sober-minded investigator to consider that 
conclusions based on etas of this value are based on less than nine 
percent of the variance in tlie criterion being accounted for by 
variance in the independent variables. It appears that this current 
concern with sophistication In statistical methodology is leading to 
an obsessive interest in numbers and, unfortunately, a lack of con- 
sideration about the questions that are being asked. 

I would contend that we are far from the point of theoretical 
testing and formulation in the area of curriculum research that is a 
late stage in any science. We are, at present, far from being able 
to state with any x^recision the objectives we are attempting to achieve 
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nor have we attempted to validate these objectives in the lives of these 
individuals with whom we are attempting to work. Hie pressures for 
publication and time limited grants have given rise to a vast array 
of "little” studies so appropriately and accurately castigated by 
Professor Leton. Such studies as two minute exercises in key pressing 
or brief verbal responses are not likely to help us much in attempting 
to improve curricula practices ahd methodology. Design and methodological 
consideration should await the formulation of meaningful questions 
which can best be ascertained by more thoughtful inquiries into the 
nature and aspects of our goals and objectives. The theorists frequently 
counter such arguments by saying^ "but look \diat interesting conclusions 
and observations my theoretical conceptualization has led us to." One 
may also counter that argument by considering what interesting questions 
and conclusions that conceptualization has perhaps led us from. 

I cannot avoid commenting on what I consider a serious omission fr<xi 
Professor Leton* s presentation. He discusses^ in sone detail^ many of 
the problems connected with educing information about important rele* 
vant interactions between kinds of students and types of curricula 
approaches. I must express my strong belief that the function of the 
teacher as an intervening ^’ariable between curriculum and the student is 
crucial for the progress of improvement in curricula research. For 
instance^ there is little doubt that various kinds of audiovisual aids 
are extremely effective in improving or accelerating the amount of 
learning students may experience. Ihere is also little doubt that 
some teachers do not function well utilizing seme of these media and, 
in fact« their performance may, in fact, suffer. I argue that the 
neglect of the teacher variable in curriculum research is a crucial 
problem and one which is not being attacked. As Professor Leton himself 
pointed out in his study with Busivell that the theoretical approaches 
en^loyed were apparently filtered out in the teacher *s system, and the 
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source of variation attributed to teachers was therefore far greater 
than the source of variation attributed to either theoretical position 
understudy. Perhaps tiiis might indicate that an important area of 
study should be the characteristics and practices of successful, voca- 
tional educators as defined by logically relevant criteria may have 
greater potential for advancement in the long run than the examination 
of various theoretical approaches to the presentation of learning 
materials. 

This preoccuxjation with theory is, at present, a possible 
stumbling block in the advancement of the practice of vocational 
education. The preoccupation with theory that is presently noted in 
the U.S. Office of Vocational Education Research Division is well 
known. Sociological and psychological conceptions are far more likely 
to receive support than are more pragmatic, and possibly more revealing 
in the long run, projects. Science does not advance by proof; science 
advances only by disproof by rejecting contentions, not by proving 
contentions. Theoretical conceptualizations, unfortunately, are far 
more apt to possess higher amounts of ego involvement and possibly be 
more subject to confirmation by subjective arithmetic than perhaps a 
more anthropological approach to the phenomenal we arc investigating. 

1 would argue that perhaps a more viable model at present would be that 
which has been described as multiple-hypothesis testing. We would 
start by observing the behavior of concern and devising a series of 
multi or competing hypotlieses which would either explain or fail to 
explain the observed behavior through experiments designed to eliminate 
completing hypothesis until only those vdiich we fail to refute remain. 

In this way we may make greater progress than by starting with a single 
unitary hypothesis with which too many studies begin. This is not a 
new idea in research; it has been called the process of converging 
operations, and it is however, an extremely difficult idea to implement. 
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It has the advantage^ however » of not binding the investigator (s) to a 
tunneltype perception of the world. Curricula research is not at the 
theoretically fornulated hypothesis testing stage. V/e need more long!** 
tudinal, descriptive studies, what happens to different kinds of people 
when exposed to different kinds of experiences and who move to different 
kinds of settings rather than rather elaborately controlled and 
sophisticated studies teasing out small differences in variation of 
behavior. 

As a final comment, perhaps our friends, Flo's, rut might have 
some values. As Professor Leton himself points out, wholesale changes 
in curriculum practices based upon little studies with insignificant 
strength association between dependent and independent variables along 
as with such fads as programmed learning, multi-media laboratories 
and other "exciting approaches to curriculum innovation" has some 
social significance. Scientific propensity to parsimony is perhaps 
resisted by the unparsimonius condition of nature that v;e confront. I'diy 
use Ocam's razor when we are trying to hack away at a jungle? The 
principle of reductionalism to which Professor Leton refers us perhaps 
is reflected more in our need to be scientific than in the nature of the 
phenomena we are attempting to observe. 

I certainly hope that my remarks will be construed as an attempt 
to present another side of the issue to which both Professor Leton and 1 
have addressed ourselves. 1 cannot argue that "I am right." 1 can 
maintain that there is more than one position upon which we might 
choose to operate. 



